Skip to content

DOC: initializer-rename doc updates and notebook/test fixes (post #2157)#2187

Open
romanlutz wants to merge 32 commits into
microsoft:mainfrom
romanlutz:romanlutz-validate-pr-2157
Open

DOC: initializer-rename doc updates and notebook/test fixes (post #2157)#2187
romanlutz wants to merge 32 commits into
microsoft:mainfrom
romanlutz:romanlutz-validate-pr-2157

Conversation

@romanlutz

@romanlutz romanlutz commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Description

Small, independent documentation and test-robustness fixes surfaced during pre-release (v1.0.0) validation after #2157. No pyrit/** production code is touched.

Docs — initializer rename (v1.0.0):

  • doc/getting_started/pyrit_conf.md: add a migration note that the default initializers were consolidated and renamed leading up to v1.0.0, so pre-v1.0.0 configs know the removed names (simple, airt, scenario_technique, objective_list) must be replaced.
  • doc/gui/0_gui.md: update a stale target auto-populate reference from the removed airt initializer to target.
  • doc/getting_started/configuration.md: correct the in-memory setup blurb — it mentioned "converter/scorer config" although no converter initializer is configured; it now describes the targets and scorers registered for selection by name or tag.

Docs — notebooks:

  • doc/code/setup/1_configuration: remove a malformed duplicate JupyText header block embedded mid-file and move the output_attack_async import into the main imports cell.
  • doc/code/targets/use_huggingface_chat_target: switch the demo model to the smaller HuggingFaceTB/SmolLM2-135M-Instruct (faster to download/run in the auto-executed integration test) and remove the try/except that swallowed model errors, so failures surface.

Tests:

  • tests/integration/scenarios/test_notebooks_scenarios.py: add a per-notebook cell-timeout override (1800s) for 3_adaptive_scenarios.ipynb, whose single cell runs many objectives across several attempts and exceeds the default 600s.

Tests and Documentation

  • Documentation and test-infrastructure only; no production code changes.
  • use_huggingface_chat_target.ipynb is auto-executed by tests/integration/targets/test_notebooks_targets.py; the smaller model keeps it within the 600s per-cell budget while still exercising a real HuggingFace target end-to-end.
  • 3_adaptive_scenarios.ipynb is auto-executed by tests/integration/scenarios/test_notebooks_scenarios.py; the new 1800s override prevents a false timeout on its long multi-attempt cell.
  • JupyText .py/.ipynb pairs are kept in sync per the repo convention.
  • Branch is merged up to date with main.

Co-authored-by: Copilot App 223556219+Copilot@users.noreply.github.com
Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448

Copilot AI added 30 commits July 12, 2026 14:36
Keep scenario DTOs typed for CLI output, migrate the custom-target example to first_message, and remove empty per-objective attacks when reconstructing sampled scenario resumes.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
Repair Python E2E target propagation and notebook examples, correct the
Responses vision payload, and isolate and harden frontend E2E validation.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
Repair scanner isolation, provider resource cleanup, Hugging Face downloads, Windows-safe output, and executable notebook examples discovered by exhaustive post-merge validation.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
Use the newly configured adversarial_chat target in the benchmark end-to-end smoke test after confirming registration and a clean 7/7 full scenario run.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
Remove the generic scorer lifecycle method and threshold-wrapper delegation while retaining Azure Content Safety's explicit resource cleanup.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
Store the audited execution results for all runnable documentation notebooks so output cells can be reviewed directly in the branch changeset.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
Retain Realtime WebSocket cleanup as a public API while moving ordinary OpenAI teardown behind a private hook and removing validation-only cleanup calls from documentation notebooks.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
…lidation branch

Resolved notebook conflicts in doc/code/converters/1_text_to_text_converters and doc/code/executor/1_single_turn by regenerating executed outputs from the merged .py sources. Verified affected tests (110 passed), full pre-commit, and notebook cell-output audit (0 errors).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
Preserve refreshed executed outputs prior to removing split-owned code
changes and merging latest main.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
Merge latest origin/main (27d20ee). Resolutions:
- 26 A-G split-owned code/test files reset to origin/main (now covered by
  PRs microsoft#2208-2214), so this branch carries no production-code duplication.
- matrix_atomic_attack_builder(.py/test) taken from main (orphaned core
  refactor does not belong on a notebook-validation branch).
- Executed notebook .ipynb outputs kept from this branch (regenerated next).

Result vs main: notebooks + notebook-source hardening + docs + frontend e2e only.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
The first TextAdaptive() cell in 3_adaptive_scenarios.ipynb runs many
objectives across several attempts and exceeds the default 600s per-cell
timeout. Give that notebook a 1800s per-cell timeout via a per-notebook
override, leaving other scenario notebooks at the 600s default.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
Sync validation branch with latest main. Resolved doc/scanner/airt.ipynb
conflict (both sides changed source) by regenerating the notebook from the
merged jupytext-paired airt.py.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
Revert 40 notebooks whose cell SOURCE is byte-identical to origin/main
(only re-executed LLM outputs / execution_count / metadata differed) plus
2 jupytext-paired .py files that differed only by a jupytext_version header
stamp. pyrit/** is identical to origin/main, so these diffs cannot reflect
any behavior change. Keeps only files with real source/doc/test changes.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
These 4 notebooks carried real .ipynb source edits (added Context Compliance
section, reworded modality-feedback objective/scorer, heading numbering,
garak encoding-doc trim) that were not reflected in their jupytext-paired
.py. Regenerated each .py from its .ipynb so the pair is consistent,
preserving each file's original jupytext_version stamp so the diff shows
only the real source change.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
Main dropped numeric prefixes from notebook H1 titles. Sync the validation
branch:
- Un-number kept notebooks (real source edits remain):
  3_image_converters ('# Image Converters'),
  8_non_llm_targets ('# Azure Blob Storage Targets').
- After un-numbering, 1_loading_datasets and 2_openai_responses_target
  differ from main only by re-executed output noise, so revert both
  pairs (.ipynb + .py) to origin/main; they drop from the changeset.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
Remove the DejaVu-Sans font_manager workaround (and the try/except unwrap)
from doc/code/converters/3_image_converters.{ipynb,py}, restoring main's
version. The underlying cause -- AddTextImageConverter defaulting to the
non-cross-platform "helvetica.ttf" -- is being fixed at the source (default
to None -> Pillow bundled default) in a separate session, after which this
notebook runs cleanly on its plain defaults with no font warning.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
…nce executor)

The branch added a "Context Compliance" section importing
ContextComplianceAttack from pyrit.executor.attack, plus its supporting
adversarial-target setup and narrative. That class no longer exists in main
(context compliance was migrated out of executor/attack), so the section is
broken against main and the change is unnecessary. Restore main's version of
doc/code/executor/1_single_turn.{ipynb,py}.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
…orkaround)

5_workflow.ipynb is explicitly CI-skipped (test_executor_notebooks.py skipped_files) because it needs a live Azure Storage Account + local AI-recruiter service. The branch had added runtime storage_available/recruiter_available guards and re-executed the notebook in an infra-less environment, which replaced main's preserved XPIAStatus.SUCCESS demonstration outputs with 'example not run: unavailable' skip messages. Main's version still resolves against current API, so restore it verbatim.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
…fix)

The validation branch's 8_modality_feedback carried a simplified 5-criteria
objective (plus a text-only profile and max_turns=8) that predated and
discarded microsoft#2188 "FIX: Preserve iterative image state in Crescendo". Restore
the origin/main version so the notebook keeps the detailed albatross
objective, 7-criteria scorer, and iterative-image-state fix.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
…aults doc)

The validation branch's garak notebook trimmed the "Aggregate techniques:
ALL / DEFAULT" markdown block that microsoft#2190 "Standardize garak.encoding
defaults" added on main. Restore the origin/main version so the DEFAULT/ALL
aggregate-technique documentation is preserved.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
…soft#2222)

The Playwright host-resolution change on this validation branch duplicates
and conflicts with PR microsoft#2222 "FIX Stabilize Playwright frontend host
resolution", which is the dedicated PR for playwright.config.ts. Revert to
origin/main so microsoft#2222 owns that file. (The e2e timing-stabilization tweaks in
api.spec.ts / flows.spec.ts are independent and left in place.)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
…ion into PR microsoft#2222)

api.spec.ts and flows.spec.ts carried e2e timing-stabilization tweaks
(explicit /api/health timeouts, a toBeVisible wait, a beforeAll setTimeout).
Per decision to consolidate all Playwright e2e stabilization into PR microsoft#2222,
revert these to origin/main here; the tweaks are folded into microsoft#2222 alongside
the playwright.config.ts host-resolution fix.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
…ch main)

Undo the verbose TargetRegistry.get_registry_singleton().instances.get(...)
pattern in round_robin_target, airt, and benchmark; restore main's canonical
OpenAIChatTarget() usage. These registry rewrites (and the entangled
validation-only bounding tweaks that referenced the registry vars) were
scaffolding added during pre-release validation and are not wanted in docs.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
Undo the validation-only auth workaround (explicit endpoint/model + Entra
get_azure_openai_auth fallback). Main's canonical bare OpenAITextEmbedding()
resolves to Entra auth when no embedding key is set; the Entra path belongs in
the environment (.env), not hardcoded in the doc notebook.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
Undo validation-only scaffolding: main lists/looks up scenarios from
import-time class registration (no init needed) and builds the target
directly via OpenAIChatTarget(), initializing late with just
LoadDefaultDatasets(). The added TargetInitializer/ScorerInitializer/
TechniqueInitializer are unused here and only add env-config fragility.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
…meout)

Restores the unbounded production-default adaptive example to match origin/main;
keeps the 1800s per-cell override in test_notebooks_scenarios.py so the auto-run
scenarios integration test has headroom instead of the default 600s.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
2_resiliency: drop unused Target/Scorer/Technique initializers (same scaffolding
reverted in 1_class_registry). 8_non_llm_targets: restore AZURE_SQL backend and the
unconditional Azure Blob example, dropping the storage-availability gating workaround.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
1_pyrit_scan: drop dynamic-port + explicit config-path scaffolding.
10_1_playwright_target: drop Windows sys.platform gating (skip-listed, not auto-run).
4_openai_video_target: drop unneeded scorer/temp-file cleanup + var extraction and the
+10313-line re-executed output committed during validation.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
Copilot AI added 2 commits July 19, 2026 18:50
Integrates microsoft#2169 (ImageResizingConverter clamp) and microsoft#2186 (Scenarios/Technique Registry alignment).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
Readers may not know what 'the initializer cleanup' refers to; frame it as the
consolidation/rename leading up to v1.0.0 so pre-v1.0.0 configs know when the old
names (simple/airt/scenario_technique/objective_list) were removed.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 52d2518d-ebe5-4888-8189-6146b6503448
@romanlutz romanlutz changed the title Pre-release validation: executed notebook outputs + validation fixes (post #2157) DOC: initializer-rename doc updates and notebook/test fixes (post #2157) Jul 20, 2026
@romanlutz
romanlutz marked this pull request as ready for review July 20, 2026 02:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants